Automated verification of a test model for a plurality of defined bdd test scenarios

ABSTRACT

A—method for automated verification of a test model for BDD test scenarios including receiving the test model to be verified; generating a model path tree using a model-based path analysis based on the test model; filtering any test step of the plurality of test steps of the paths in the model path tree; d. identifying any test step with a missing alternative test step in another alternative path of the plurality of paths in the model path tree using machine learning based on the model path tree; adding at least one missing alternative test step to the test model after identification of at least one test step, resulting in at least one alternative path; and providing the identified test step, the identified at least one missing alternative test step, the at least one alternative path and/or the at least one alternative BDD test scenario as output data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to EP Application No. 21168972.4,having a filing date of Apr. 16, 2021, the entire contents of which arehereby incorporated by reference.

FIELD OF TECHNOLOGY

The following relates to a computer-implemented method for automatedverification of a test model for a plurality of defined BDD testscenarios. Further, the following relates to a corresponding computingunit and computer program product.

BACKGROUND

In recent years, Behavior Driven Development (BDD) has emerged as anagile software development approach for the specification and executionof automated acceptance tests of software programs. BDD aims to simplifyTest-Driven Development (TDD). TDD is a software development methodologywhich essentially states that for each unit of software, a softwaredeveloper must define specific test sets for the unit first, thenimplement the unit and finally verify that the implementation of theunit makes the tests succeed. BDD combines Test-Driven Development(TDD), Object-Oriented Analysis (OOA), Object-Oriented Design (OOD) andDomain-Driven Design (DDD) to provide a unified language and approachfor handling such a software development process from requirementsanalysis to implementation.

BDD is largely facilitated through the use of a simple domain-specificlanguage (DSL) using natural language constructs (e.g., English-likesentences) that can express the behavior and the expected outcomes ofthe software. This ‘ubiquitous language’ can be understood and jointlyused by quality managers, domain experts, software developers andcustomers. BDD employs a semi-formal format for behavioral specificationof the software, which is borrowed from user story specifications fromthe field of object-oriented analysis and design.

To this end, the behavior of each software unit is decomposed intoso-called scenarios, each scenario testing one individual aspect of thesoftware. Each scenario is in turn divided into steps, which describe adesired outcome of the respective aspect of the software starting fromgiven initial conditions and running through predefined events. Eachscenario with its steps is formulated as a natural language script,which can later be translated into executable specifications as well asexecutable test scripts in an automated way. The executable test scriptscan then be executed as automated tests for testing the software for itscorrect implementation. The software requirements within the testscripts are usually written in “given-when-then” sentences based on theubiquitous language of domain-driven design. This is intended tofacilitate the transition between the language used to define thedomain-driven requirements and the programming language used toimplement them.

One test automation framework widely used for automated acceptance testswritten in BDD style is called Cucumber. Most test automationframeworks, including Cucumber, are based on the Gherkin notation as theDSL. Cucumber comprises a plain language parser for the Gherkinnotation. The desired behavior of the software is formulated withinGherkin in a declarative way:

-   -   GIVEN (precondition/initial conditions) . . .    -   WHEN (event/action/trigger) . . .    -   THEN (effect to be observed/system response) . . .

Such descriptive languages are semi-formal with the capital words(GIVEN, WHEN, THEN) serving as pre-designated keywords. Due to thesimple grammar and the natural language keywords, the BDD requirementscan be understood and manually executed by technical testers. Cucumberruns through these keywords and processes them step by step, therebymapping every non-capital phrase following these keywords to aparameterized function call. Traditionally, Ruby scripts were used forthis purpose within Cucumber, which replace the test steps by automatedprogram calls and thus make the BDD description automaticallyexecutable. However, Cucumber now supports a variety of differentprogramming languages through various implementations, including Javaand C#.

BDD is easy to understand and straightforward to implemented. However,in large and complex use cases, the approach with its textual andmanually created scenarios may lack the manageability to handle a largenumber of scenarios and complex tests sets while ensuring completenessand consistency. For the development of complex systems, approaches likeModel Based Testing (MBT) or Keyword Based Testing (KBT) are often seenas more appropriate. MBT approaches allow reviewing and verifying thecompleteness and consistency of even complex test scenarios using avisual representation of the scenarios, e.g., using diagrams in UnifiedModelling Language (UML). However, MBT has to be individually embeddedinto the existing development and test process for each softwarecomponent.

SUMMARY

An aspect relates to a computer-implemented method for automatedverification of a test model for a plurality of defined BDD testscenarios in an efficient and reliable manner.

This problem is according to one aspect of embodiments of the inventionsolved by a com-puter-implemented method for automated verification of atest model for a plurality of Behavior Driven Development, BDD, testscenarios, the method comprising:

-   a. receiving the test model to be verified;-   b. generating a model path tree using a model-based path analysis    based on the test model; wherein    -   the model path tree comprises a plurality of paths, each path        assigned to a respective BDD test scenario of the plurality of        BDD test scenarios; wherein    -   each path comprises a plurality of test steps and respective        intersection information;-   c. filtering any test step of the plurality of test steps of the    paths in the model path tree to be filtered regarding their impact    on test coverage or another filter criterion using natural language    processing based on the model path tree;-   d. identifying any test step with a missing alternative test step in    another alternative path of the plurality of paths in the model path    tree using machine learning based on the model path tree;-   e. adding at least one missing alternative test step to the test    model after identification of at least one test step, resulting in    at least one alternative path; wherein    -   the at least one alternative path is assigned to at least one        alternative BDD test scenario; and-   f. providing the identified test step, the identified at least one    missing alternative test step, the at least one alternative path    and/or the at least one alternative BDD test scenario as output    data.

Accordingly, embodiments of the invention are directed to acomputer-implemented method for automated verification of a test modelfor a plurality of BDD test scenarios. Thereby, the test model serves asinput data for the method steps and can be derived in different ways.For example, the test model can be derived from BDD scenarios based onBDD step definitions and similarity analysis using Neuro LinguisticProgramming and machine learning mechanisms. Alternatively, otherapproaches can be applied.

In a first step, the model-based path analysis is applied on theprovided test model as input data to generate a model path tree. Inother words, the test model is processed into a tree structure withpaths comprising test steps and respective intersection information.Thereby, the intersection information can comprise paths like branch andmerge locations. Each path is associated with a respective BDD testscenario.

In a next step, natural language processing is applied on the model pathtree to filter test steps using a filter criterion. In other words,irrelevant or not essential test steps are removed from the model pathtree. This step is essential to reduce the amount of test steps in themodel path tree for step d., according to which identification. Thisstep results in zero to at least one filtered test steps. In otherwords, in this case, the test model is already complete.

In the next step machine learning (e.g., a machine learning model suchas a pre-learned or trained neural network, learned on previous projectsor project examples) is applied on the model path tree to identify teststeps with missing alternative test steps in another alternative path ofthe model path tree. In other words, the model path tree, comprising theplurality of the paths and its test steps, is analyzed to identifyadditional branches impacting the test coverage or any otheroptimization criteria. This step results in zero to at least oneidentified test step with missing alternatives, zero to at least onealternative path and/or zero to at least one alternative BDD testscenario.

According to which, at least one missing alternative test step is addedto the test model after identification of one or more test steps,resulting in one or more alternative path. Thus, the model path tree isextended. The extension takes place.

Hence, the output data is provided in the last step after identificationof at least one missing test step. Accordingly, having identified one ormore missing test steps, in this case, the output data is provided.

In the event that no or zero test steps are missing and thus cannot beidentified in step d., in this case, other output data can be provided,for example a notification no test step is missing.

The advantage of embodiments of the present invention are that missingtest steps and hence associated alternative BDD test scenarios can beefficiently determined. Embodiments of the present invention are alsoapplicable on huge amounts of data, hence big data, due to thepre-filtering. The identification ensures the consistency and thecompleteness of BDD test scenarios.

Moreover, after identification, the alternative BDD test scenarios canbe handled in an efficient and reliable manner, the received test modelreceived in step a. can be adapted depending on the identified missingtest steps and alternative BDD test scenarios.

The adapted test model can serve as basis or input for the generation ofthe executable test scripts for the BDD test scenarios. This way, thespecification and execution of automated acceptance tests of softwareprograms can be significantly improved.

To the contrary, according to prior art, known approaches are solelymanual and are only applicable for reasonably small projects.

Moreover, embodiments of the present invention relate to Behavior-DrivenDevelopment (BDD) and combines traditional BDD advantages with ModelBased Testing (MBT) for improved convenience and automatization in caseof complex software packages

According to an aspect, the method further comprises at least one stepof the following steps:

-   -   transmitting the output data to a computing unit;    -   storing the output data in a volatile or non-volatile storage        unit;    -   displaying the output data on a display unit of a computing unit        to a user;    -   providing the output data via an interface of a computing unit        to a user;    -   extending the test model with the alternative BDD test scenario;        and    -   generating executable test scripts for the BDD test scenarios        based on the test model and/or extended test model.

Accordingly, the input data, data of intermediate method steps and/orresulting output data can be further handled. The output data is theidentified test step and/or the alternative BDD test scenario. One ormore actions can be performed. The action can be equally referred to asmeasure.

These actions can be performed by one or more technical units, such ascomputing unit or robot unit. The actions can be performed gradually orsimultaneously. Actions include e.g., storing and processing steps. Theadvantage is that appropriate actions can be performed in a timelymanner.

In an embodiment, the identified alternative BDD test scenarios areconsidered in the test model which serves as input or basis for thegeneration of executable test scripts for the BDD test scenarios, seefurther above. Hence, in other words, the test model is extended withthe alternative BDD test scenarios.

For example, according to a use case, the generated output data providessignificant input to a test manager or analyst to extend and optimizeexisting BDD scenarios to ensure adequate test coverage.

A further aspect of embodiments of the invention is a computing unite.g., robot unit or another autonomous unit.

The unit may be realized as any devices, or any means, for computing,for executing a software, an app, or an algorithm. For example, the unitmay consist of or comprise a central processing unit (CPU) and/or amemory operatively connected to the CPU. The unit may also comprise anarray of CPUs, an array of graphical processing units (GPUs), at leastone application-specific integrated circuit (ASIC), at least onefield-programmable gate array, or any combination of the foregoing. Theunit may comprise at least one module which in turn may comprisesoftware and/or hardware. Some, or even all, modules of the unit may beimplemented by a cloud computing platform.

A further aspect of embodiments of the invention is a computer programproduct (non-transitory computer readable storage medium havinginstructions, which when executed by a processor, perform actions).

BRIEF DESCRIPTION

Some of the embodiments will be described in detail, with reference tothe following figures, wherein like designations denote like members,wherein:

FIG. 1 shows a schematic diagram of the method according to embodimentsof the invention; and

FIG. 2 shows a schematic representation of an alternative path in themodel path tree according to embodiments of the invention.

DETAILED DESCRIPTION

FIG. 1 illustrates a flowchart of the method according to embodiments ofthe invention with the method steps S1 to S6.

Providing the Input Data S1

First, the input data in form of the test model is provided S1.

Transformation of the test model into the model path tree S2

The model-based representation of the BDD test scenarios allows toutilize model-based analysis approaches, like path analysis. Themodel-based path analysis generates a tree structure describing thestructure of the defined BDD test scenarios and the differentalternative paths 10, S2. Thereby, the paths comprise test steps 12.

Filtering Test Steps S3

Further, NLP is used to filter out activities in the model path treewhich are not relevant in view of e.g., test coverage.

Further, the number of missing alternative paths (additionally requiredBDD scenarios) can be estimated using specific metrics, like pathcomplexity, etc.

Identification of Missing Test Steps S4

Machine learning is used to compare the paths of the model path treewith previous project examples and the resulting impact on testcoverage.

As illustrated in FIG. 2, one path comprises the test step “I click OK”,whereas the other path does not comprise the test step “I click OK”.Accordingly, the missing test step is existent in the other alternativepath. This identified test step and alternative path is assigned to thealternative BDD test scenario S5.

Providing the Output Data S6

The output, the alternative BDD test scenario, of the method can be usedto extend and optimize the BDD test model to achieve the desired testcoverage.

Although the present invention has been disclosed in the form ofpreferred embodiments and variations thereon, it will be understood thatnumerous additional modifications and variations could be made theretowithout departing from the scope of the invention.

For the sake of clarity, it is to be understood that the use of “a” or“an” throughout this application does not exclude a plurality, and“comprising” does not exclude other steps or elements.

1. A computer-implemented method for automated verification of a testmodel for a plurality of Behavior Driven Development (BDD) testscenarios, the method comprising: a. receiving the test model to beverified; b. generating a model path tree using a model-based pathanalysis based on the test model; wherein the model path tree comprisesa plurality of paths, each path assigned to a respective BDD testscenario of the plurality of BDD test scenarios; wherein each pathcomprises a plurality of test steps and respective intersectioninformation; c. filtering any test step of the plurality of test stepsof the paths in the model path tree to be filtered regarding theirimpact on test coverage or another filter criterion using naturallanguage processing based on the model path tree; d. identifying anytest step with a missing alternative test step in another alternativepath of the plurality of paths in the model path tree using machinelearning based on the model path tree; e. adding at least one missingalternative test step to the test model after identification of at leastone test step, resulting in at least one alternative path; wherein theat least one alternative path is assigned to at least one alternativeBDD test scenario; and f. providing the identified test step, theidentified at least one missing alternative test step, the at least onealternative path and/or the at least one alternative BDD test scenarioas output data.
 2. The computer-implemented method according to claim 1,wherein the method further comprises: transmitting the output data to acomputing unit; displaying the output data on a display unit of acomputing unit to a user; providing the output data via an interface ofa computing unit to a user; extending the test model with thealternative BDD test scenario; and generating executable test scriptsfor the BDD test scenarios based on the test model and/or extended testmodel.
 3. A computing unit for performing the method steps according toclaim
 1. 4. A computer program product, comprising a computer readablehardware storage device having computer readable program code storedtherein, said program code executable by a processor of a computersystem to implement a method according to claim 1, when the computerprogram product is running on a computer.